This PR implements a comprehensive hybrid retrieval system that combines dense semantic search with sparse keyword search, followed by intelligent reranking to improve retrieval accuracy and robustness.

Overview

The hybrid retrieval system addresses limitations of pure dense vector search by incorporating keyword-based matching and multi-signal reranking. This approach significantly improves retrieval performance, especially for queries requiring exact term matches or domain-specific terminology.

Key Features

🔍 Hybrid Search Function

from src.retrieval.hybrid_search import hybrid_search

# Execute hybrid search with configurable parameters
results = hybrid_search(
    query="existential meaning of life",
    top_k_dense=5,      # Dense semantic results
    top_k_sparse=20     # Sparse keyword results
)

# Results include detailed scoring breakdown
for result in results:
    print(f"Score: {result['relevance_score']:.4f}")
    print(f"Source: {result['source']}")  # 'dense', 'sparse', 'both'
    breakdown = result['score_breakdown']
    print(f"Dense: {breakdown['normalized_dense']:.3f}")
    print(f"Sparse: {breakdown['normalized_sparse']:.3f}")
    print(f"Overlap: {breakdown['overlap_score']:.3f}")

📊 Intelligent Reranking Algorithm

The system combines multiple relevance signals with optimized weights:

• Dense semantic score (weight: 0.5) - Vector similarity using existing embeddings
• Sparse keyword score (weight: 0.3) - TF-IDF based term matching
• Lexical overlap ratio (weight: 0.2) - Direct query-document term intersection

🏗️ Dual Index Architecture

• Dense index: philosophy-rag - Existing managed embeddings for semantic search
• Sparse index: philosophy-rag-sparse - New TF-IDF weighted sparse vectors for keyword search
• Consistent chunk IDs: Same identifiers across both indexes for proper result merging

🔧 Advanced Sparse Vector Construction

# TF-IDF formula: (1 + log(tf)) * log((N + 1) / (df + 1)) + 1
# Tokenization: lowercase, alphanumeric split, stopword filtering, min length 2
# Vocabulary management with persistent storage

Technical Implementation

New Components

• src/storage/sparse_store.py - Vocabulary management, TF-IDF calculation, sparse vector operations
• src/retrieval/hybrid_search.py - Result merging, reranking, and hybrid search orchestration
• data/vocab.json - Token-to-integer mapping for sparse vectors
• data/df.json - Document frequencies for IDF calculations

Enhanced Ingestion Pipeline

The ingestion process now creates both dense and sparse representations:

# Updated ingestion workflow
chunks = chunk_document(pdf_content, metadata)

# Assign consistent IDs for both indexes
for chunk in chunks:
    chunk['id'] = str(uuid.uuid4())

# Store in both indexes
store_vectors(chunks)                    # Dense vectors
sparse_store.upsert_sparse_vectors(chunks)  # Sparse vectors

Graceful Degradation

• Falls back to dense-only search when sparse index unavailable
• Falls back to sparse-only search when dense index unavailable
• Maintains consistent result format regardless of available modalities

Usage Examples

Basic Hybrid Search

# Test both semantic and hybrid search
python src/scripts/test_search.py

Document Ingestion with Hybrid Indexing

# Ingest documents creating both dense and sparse indexes
python src/scripts/ingest_documents.py

Performance Benefits

1. Improved Recall: Sparse search captures exact term matches missed by dense search
2. Better Precision: Dense search provides semantic understanding beyond keyword matching
3. Robust Ranking: Multi-signal reranking reduces false positives and improves relevance
4. Fault Tolerance: Graceful degradation ensures system availability

Backward Compatibility

All existing functionality remains unchanged:

• Existing semantic search continues to work identically
• Web interface and API endpoints unaffected
• Current ingestion processes enhanced but not breaking

This implementation provides a significant upgrade to retrieval quality while maintaining full backward compatibility with existing workflows.

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.